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This research aims to build a general framework for choosing the most 
appropriate set of criteria for recruiting student as a research assistant in a 
university research project. University researchers could benefit from such a 
framework because it could optimize the costs of research while also 
enhancing students research skills. In the same time, it is also essential that the 
quality of research ought to measure up to the grants provided by the 
university. Nevertheless, it is a challenging problem for many research 
supervisors in the selection of qualified research assistants. In this paper, we 
attempted to resolve this problem by building a general framework for 
selecting the appropriate criteria in the evaluation of student performance. We 
explored earlier studies on the proposed evaluation criteria of the research 
assistant and identified 47 most impactful criteria criteria. We obtained 
experts in engineering and information technology fields from two 
universities to answer questionnaires to identify their commonly used criteria 


for grant research assistant (GRA). Then, all the identified criteria were 
evaluated using the fuzzy delphi method (FDM) for finding the best fitting 
criteria which resulted in 16 most impactful criteria. 
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1. INTRODUCTION 

The main aim of this paper is to propose a general framework for the selection of the appropriate 
criteria for evaluating student performance for a grant research assistant (GRA) in the IT and engineering 
departments at universities. Hence, it is essential to review the related theories, variables, and techniques that 
could be of support to the main aim of this research. The knowledge possessed by students could be defined 
as the skills and information acquired from learning [1]. Thus, one of the primary purposes of developing 
students’ skills is to prepare them for job placement once they graduate. Several work tasks could be 
allocated to students based on their current skills while studying at a university. 

It is undeniable that many universities focus on improving the skills and knowledge of students to 
meet the needs of the industry with human potentials [2]. The transferred skills in classrooms must have an 
impact on students in order to develop their knowledge in fields, such as doing researches and analytical 
work [3]. Away from work settings, this research set to emphasize the importance of developing and 
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enhancing student research skills at the university level in order to support and improve university research 
outputs [4], [5]. This is achieved through the offering of the GRA job scheme in universities. 

There are numerous variables available to assess the research skills of GRA [6], [7]. However, the 
selection of GRA is problematic because these research skills would overlap or intersect within matrices such 
as the evaluation variables, structure of research group, and the research proprieties. Therefore, computer- 
based methods are recommended to assist in the process of GRA selection decision making. A selection 
process must rely on several and interconnected evaluation attributes. The multi-criteria decision making 
(MCDM) would be a practical approach for effective selection of GRA based on different variables, 
attributes, and objectives. 

It is commonly and practically known that all the members of any evaluation committee would not 
share a similar view when judging a problem. A panel of experts may come to a disagreement caused by 
different opinions on the rating of the alternatives or the merit of the criteria. Arriving at the best resolution 
despite such differences becomes a significant issue in group-based decision making. If experts realize that 
using a numeric scale for expressing their opinions is convenient, it would be useful to consider averaging the 
scores as a simple way for aggregating conflicting assessments. Besides, if the opinions of the members of a 
group do not carry the same weight, then it would be essential to implement a weighted averaging scale to 
specify their relative importance. However, to solve an IT or engineering evaluation problem, it is more 
important to arrive at the right level of consensus among the experts; through encouraging them to reconsider 
their assessments rather than aggregate their scores. This is the core of the delphi method. The delphi method, 
having a repetitive procedure, aims at making various subjective opinions converge into more widely 
acceptable viewpoints [8], [9]. The difficulty faced by any research supervisor in selecting a GRA could be 
attributed to several reasons [10]-[13]: i) the variety of evaluation criteria and characteristics (there is no 
standard for the GRA evaluation and selection criteria), ii) the process of assessing the skills of researcher 
performance due to the different type of research activities. 

In this case, a supervisor would face difficulties in creating a research group with balanced 
individual and group work skills. In order to do so, a supervisor must evaluate such skills by employing 
useful variables. There is a large number of evaluation variables that can be used in this respect. However, 
this large number makes it harder for supervisors to pinpoint the most relevant variables to evaluate the 
research skills performance based on different characteristics. To solve this problem, we explored earlier 
studies pertaining to the evaluation and selection of GRA to identify the commonly used evaluation criteria. 
The criteria are then distributed to experts to evaluate followed by the application of the fuzzy delphi 
program. 


2. RESEARCH METHOD 

There are two major phases in this research. First is the identification of criteria through literatures 
reviews. The second phase is the evaluation of the criteria through experts’ opinion which are implemented 
using the fuzzy delphi method (FDM). 


2.1. Criteria identification 

Literatures related to research skills are important primary sources to identify criteria that could be 
adopted to assess the GRA skills. There are numerous skills which commonly appeared and proposed in 
literatures as important research skills. These skills could be grouped into corresponding category. Every 
group would include a collection of criteria to facilitate the process of mathematical calculations [14], [15]. 


2.2. Implementation of the fuzzy delphi method 

Such large number of criteria extracted from the literature review would need to be evaluated using 
the delphi fuzzy to achieve the consensus of experts on the most useful criteria. In other words, the criteria 
would be examined and tested in the interviews with experts. Figure | presents the flow of steps involved in 
the implementation of the FDM to determine the suitable evaluation criteria of GRA. The steps are further 
explained in sections 2.2.1 to 2.2.4. 


2.2.1. Selection of the experts 

In this study, experts are defined as researchers who have wide research experience and supervises 
many postgraduate students and GRA. In terms of the numbers of experts for the study, a consensus has to be 
reached in this regards. Previous researchers suggested that the number of experts ranges between 3 to 25 as 
an optimal number in Delphi method [16]-[20]. Nevertheless, a precondition to the experts is that they must 
have the aptitude to process information and give decisions. In this study, we chose 23 experts among 


Int J Artif Intell, Vol. 10, No. 4, December 2021: 839 - 846 


Int J Artif Intell ISSN: 2252-8938 O 841 


professors, associate professors, and lecturers of the IT and engineering departments in two participating 
universities; the University of Anbar and the University of Technology in Baghdad. 


Phase 1: List of criteria identified through literature review of previous related studies 


| Phase 2: Implementation of the Fuzzy Delphi Method for selection of best criteria 
C > >» 4 : : 


Selection of Questionnaire z Fuzzy Delphi 
experts development Data collection Data Analysis 


Figure 1. Flowchart of FDM implementation for GRA criteria evaluation 


2.2.2. Development of questionnaire 

We developed a questionnaire to collect the data (opinions of the experts) that are grounded on the 
criteria inferred from the analysis and combination of existing guidelines. Rahim et al. [21] suggested that in 
the digital age, researchers may place the questionnaire online to avoid delay and burden. Hence, the 
questionaires were built and distributed via the use of google form online survey. The questionnaires 
comprises of two parts; part one is related to the personal information of the expert and part two contains the 
list of identified criterias to be scored. The assessment was done by using the fuzzy likert with five scale 
response: strongly agree, agree, neutral, disagree, and strongly disagree. 


2.2.3. Data collection 

In this questionnaires, an electronic form of answers was presented and submitted to the selected 
group of professors, assistant professors and professors who have long experience in supervising students in 
the universities. The responses for the questionnaires were collected through google forms and downloaded 
as microsoft excel file for ease of analysis, which involved finding the average of threshold value, the 
average percentage of expert’ consensus and average fuzzy score. It is suggested that maximum and 
minimum method uses a cumulative frequency distribution and fuzzy scoring in order to deal with the 
opinion of experts with respect to the fuzzy numbers resulting from the FDM [22], [23]. 


2.2.4. Fuzzy delphi data analysis 
The FDM is used to determine the best type of criteria and to set the type of factors appertaining to 
this study. The following are the steps performed in fuzzy delphi data analysis: 
— Convert the Linguistic variables to triangular fuzzy numbers. The linguistic variables are for weighting 
the agreement of the experts. Table | as shown in the process of linguistic variables for weighting the 
agreement of the experts [19], [24]. 


Table 1. Linguistic variables of the agreement 


Linguistic variables Fuzzy likert 
Strongly Disagree (1) 0 0 0.2 
Disagree (2) 0 0.2 0.4 
Neutral (3) 0.2 0.4 0.6 
Agree (4) 0.4 0.6 0.8 


Strongly Agree (5) 0.6 0.8 1 


- Calculate the average value based on the total of number of each item and then divided by the number 
of experts [25]. 
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— Calculate the distance between two fuzzy numbers by measuring the deviation between the average 
fuzzy evaluation data and the experts' evaluation as shown in (1). Data vertex method used to calculate 
the distance, which is the threshold value (d) of the two (2) fuzzy numbers m = (m1,m2, m3) and n = 
(n1,n2,n3), which are then averaged for all the results. The last average represents the total threshold 
value (d) for that particular criteria. 


1 
d(m,n) = iF [(m, — 1)? + (mz — nz)? + (m3 — N3)*)] (1) 


— Calculate the percentage for each item, when the threshold value (response) for each criteria with d<= 
0.2. The data analysis is based on the triangular fuzzy number where it aims to get threshold value (d). 
Therefore, the first requirement to be followed is threshold value (d) must be less or equal to 0.2. 
Percentages for certain item if reach an agreement of experts exceeding 75.0%, then this item is 
accepted. Instead, if it is less than 75.0%, it means that this item need to be rejected. 

— The average fuzzy score was determined based on the value of a-cut, which is 0.5. If the average fuzzy 
score (A) is more than or equal to 0.5, the elements are regarded as has achieved the consensus of the 
experts. The formula used for defuzzification in (2): 


1 
A = 3 (im + mz + ms) (2) 


3. RESULTS AND DISCUSSION 

In this study, such findings that represent the FDM and the final set of criteria gathered from the 
literature review. This is in order to formulate the questionnaire and then select the suitable criteria. To do so, 
there are numerous methods for the identification of the relationship among various the criteria. This 
identification is based on the characteristics of such criteria. 


3.1. Identified criteria 

The results of the student classification process are presented in Table 2. Collected from literature 
review, the criteria are grouped into four categories; namely Human Behavior, Methodology skills, Mental 
and Personal skills. These categories were derived from relevant literatures and suggested by experts in a 
close format. 


3.2. Data analysis using fuzzy delphi 

Table 3 shows the results of the accepted criteria extracted from the FDM. Consistency among the 
results of each group of criteria does exist and demonstrated through the results. These accepted criteria 
where its percentage is greater than or equal to 75%, were divided into coherent categories. This process is to 
facilitated the understanding and mathematical operations in the future works. The percentage attained for 
each criteria is approved on the basis of this ratio, as described in the steps of the fuzzy delphi analysis. Other 
analysis steps were carried out in the fuzzy delphi phase along with drafting abbreviations for each element 
to facilitate the work of tables and reduce their size. 

Two Iraqi universities were selected as a case study to examine GRA skills selection, in particular, 
the lecturers from Department of IT, the Department of Computer Science and the Departments of 
Engineering in the University of Anbar and the University of Technology in Baghdad. The assessment of the 
criteria of GRA skills were done using the five-scale fuzzy likert responses: strongly agree, agree, neutral, 
disagree, strongly Disagree; each with a score of 5, 4, 3, 2, 1 respectively. The following sections and sub- 
sections elaborates on the evaluation and selection of the initial assessment decision matrix. This is in order 
to have an ideal decision matrix and accordingly exclude dimensions and elements having no impact on the 
evaluation and selection of GRA. Nevertheless, it is noteworthy that the FDM is used to get an expert 
agreement on the pillars and the elements of the said decision matrix. The findings of the experts' consensus 
on the formulation of the proposed decision matrix are summarized in Table 4, comprising of the average of 
threshold value, the average percentage of expert’ consensus and average fuzzy score. Red colored fonts 
represents the results that are rejected because of the average percentage score that is below 75%, as 
explained in the previous section. 
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Table 2. Categorization of the identified criteria for assessing GRA skills 


No. Criteria Categories 
1. Adapt to changing technology Human Behaviour 
2. Leadership ability Human Behaviour 
36 Better understand myself Human Behaviour 
4. Be ready to be a good citizen Human Behaviour 
Ds Relate well to people of different races/cultures Human Behaviour 
6. Strengthen interpersonal relationship skills Human Behaviour 
D3 Understand cultural differences Human Behaviour 
8. Hypotheses design Methodology Skills 
9. Know literature of merit in field Methodology Skills 
10. Cope with conflict Methodology Skills 
11. Appreciate artistic and creative expressions Methodology Skills 
12. Carry out research Methodology Skills 
13. Skill of conducting literature review Methodology Skills 
14. Understanding of scientific findings Methodology Skills 
15. Capable of documenting the research Methodology Skills 
16. Capability to gather data within a group Methodology Skills 
17. Acquire information on my own Methodology Skills 
18. Place current issues in historical context Methodology Skills 
19. Possess clear career goals Methodology Skills 
20. Researching skills (acquiring information). Methodology Skills 
21. Research presentation skills Methodology Skills 
22. Understandinf of ethical implications Methodology Skills 
233 Understanding of research concepts Methodology Skills 
24. Think logically about complex material Mental 
25: Tolerate ambiguity Mental 
26. Understanding of math concepts Mental 
21. Analyze literature critically Personal Skills 
28. Approach problems creatively Personal Skills 
29. Basic skills Personal Skills 
30. Communication skills (i.e. work as part of team) Personal Skills 
31. Computer skills Personal Skills 
32. Capability to obtain data Personal Skills 
33. Speak effectively Personal Skills 
34. Write effectively Personal Skills 
35. Develop intellectual curiosity Personal Skills 
36. Listen effectively Personal Skills 
31. Maintain openness to new ideas Personal Skills 
38. Skills of data analysis Personal Skills 
39. Skills of data collection Personal Skills 
40. Skills of data testing Personal Skills 
41. English proficiency skills Personal Skills 
42. Solve problems independently Personal Skills 
43. Synthesize and use info from diverse sources Personal Skills 
44. Use of foreign language Personal Skills 
45. Use statistics or math formulas Personal Skills 
46. Utilize computer skills Personal Skills 
47. Work as part of a team Personal Skills 


Table 3. Percentages achieved by different criteria under different categories 


No. Accepted criteria Percentage of each item 
Human Behaviour 
1. Act as a leader 91% 
2s Prepare to be good citizen 95% 
3. Relate well to people of diff. races/culture 95% 
Methodology skills 
1 Acquire info on my own 86% 
2: Hypotheses design 86% 
3. Know literature of merit in field 82% 
4 Skill of conduct the literature review. 95% 
5 Understand scientific findings 82% 
Mental 
1. Appreciate artistic & creative experiences 771% 
2s Think logically about complex material 711% 
3. Tolerate ambiguity 86% 
4. Understand math concepts 771% 
Personal skills 
1. Basic skills (reading, writing and speaking) 771% 
25 Computer skills 91% 
3. Speak effectively 95% 
4. Write effectively 82% 
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Table 4. Fuzzy delphi results for criteria of GRA 


Triangular Fuzzy Numbers 


No. Criteria Average threshold Average percentage Average fuzzy Result 
value (d) of expert consensus score (A) 

1. Capability to obtain data 0.2 45% 0.627 Rejected 
2. Capability to document the research 0.2 59% 0.612 Rejected 
3. Capability to gather data within a group 0.2 45% 0.591 Rejected 
4. Acquire info on my own 0.2 86% 0.500 Accept 
5. Act as a leader 0.2 91% 0.518 Accept 
6. Adapt to changing technology 0.2 45% 0.609 Rejected 
7. Analyze literature critically 0.2 59% 0.612 Rejected 
8. Appreciate artistic and creative expressions 0.1 711% 0.582 Accept 
9. Approach problems creatively 0.2 59% 0.555 Rejected 
10. Basic skills 0.2 711% 0.639 Accept 
11. Better understand myself 0.2 64% 0.555 Rejected 
12. Carry out research 0.2 59% 0.636 Rejected 
13. Communication skills 0.2 36% 0.609 Rejected 
14. Computer skills 0.2 91% 0.673 Accept 
15. Cope with conflict 0.2 59% 0.564 Rejected 
16. Develop intellectual curiosity 0.2 41% 0.582 Rejected 
17. Hypotheses design 0.2 86% 0.564 Accept 
18. Know literature of merit in field 0.2 82% 0.509 Accept 
19. Listen effectively 0.2 36% 0.573 Rejected 
20. Maintain openness to new ideas 0.2 55% 0.555 Rejected 
21. Place current issues in historical context 0.2 55% 0.545 Rejected 
22. Possess clear career goals 0.2 59% 0.573 Rejected 
23: Prepare to be a good citizen 0.2 95% 0.527 Accept 
24. Relate well to people of different races/culture 0.2 95% 0.591 Accept 
25. Researching skills (acquiring information) 0.2 45% 0.609 Rejected 
26. Skill of conduct the literature review. 0.1 95% 0.600 Accept 
27. Skills of data analysis 0.2 41% 0.630 Rejected 
28. Skills of data collecting 0.1 59% 0.609 Rejected 
29. Skills of data testing 0.2 55% 0.655 Rejected 
30. English proficiency skills 0.2 55% 0.603 Rejected 
31. Research presentation skills 0.2 41% 0.585 Rejected 
32. Solve problems independently 0.2 45% 0.612 Rejected 
33. Speak effectively 0.3 95% 0.524 Accept 
34. Strengthen interpersonal relationship skills 0.2 50% 0.564 Rejected 
35. Synthesize & use info from diverse sources 0.2 59% 0.545 Rejected 
36. Think logically about complex material 0.1 711% 0.564 Accept 
37. Tolerate ambiguity 0.2 86% 0.518 Accept 
38. Understanding of cultural differences 0.2 45% 0.555 Rejected 
39. Understanding of ethical implications 0.2 55% 0.609 Rejected 
40. Understanding of math concepts 0.2 771% 0.600 Accept 
41. Understanding of scientific findings 0.1 82% 0.609 Accept 
42. Understanding of research concepts 0.2 55% 0.609 Rejected 
43. Use of foreign language 0.2 41% 0.548 Rejected 
44. Use statistics or math formulas 0.2 59% 0.555 Rejected 
45. Utilize computer skills 0.2 41% 0.594 Rejected 
46. Work as part of a team 0.3 50% 0.536 Rejected 
47. Write effectively 0.2 82% 0.539 Accept 


4. CONCLUSION 

This paper proposed a general framework for the selection of most fitting criteria for GRA based on 
experts’ evaluations. There were 47 initial criteria obtained through literature review. There were 23 experts 
from IT and engineering related departmens at two universities; University of Anbar and the University of 
Technology in Baghdad were presented with these 47 criteria in a questionaire form to score using five scale 
fuzzy likert response. The expert's responses on these criteria were evaluated using the FDM, where in turn 
16 criteria that met the condition for acceptance were selected as the best impactful criteria. Future works 
will attempts to assess these 16 standards criteria with MCDM technique in order to select the best candidate 
for GRA. 
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